Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Differential Diffusion to Auraflow. #9597

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

satani99
Copy link
Contributor

@satani99 satani99 commented Oct 6, 2024

What does this PR do?

Fixes partially #8924

Before submitting

How to test

import torch
from diffusers.utils import load_image
from pipeline_aura_flow_differential_img2img import AuraFlowDifferentialImg2ImgPipeline
pipe = AuraFlowDifferentialImg2ImgPipeline.from_pretrained(
    "fal/AuraFlow", torch_dtype=torch.float32
)
pipe.enable_sequential_cpu_offload()
source_image = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png"
)
map = load_image(
    "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask_2.png"
)
prompt = "a green pear"
negative_prompt = "blurry"
image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    image=source_image,
    num_inference_steps=28,
    guidance_scale=4.5,
    strength=1.0,
    map=map,
).images[0]

Who can review?

@a-r-r-o-w @DN6

@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Oct 7, 2024

cc @asomoza or @a-r-r-o-w for a review!

Copy link
Member

@a-r-r-o-w a-r-r-o-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this! Just some minor comments, and could you post your results here too?

f" {negative_prompt_attention_mask.shape}."
)

def encode_prompt(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a # Copied from here?

max_sequence_length: int = 256,
output_type: Optional[str] = "pil",
return_dict: bool = True,
map: PipelineImageInput = None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think docstring of this is missing

def denoising_value_valid(dnv):
return isinstance(dnv, float) and 0 < dnv < 1

# sigmas = np.linspace(1.0, 1 / num_inference_steps, num_inference_steps)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# sigmas = np.linspace(1.0, 1 / num_inference_steps, num_inference_steps)

Comment on lines +624 to +629
# timesteps, num_inference_steps = self.get_timesteps(
# num_inference_steps,
# strength,
# device,
# denoising_start=self.denoising_start if denoising_value_valid(self.denoising_start) else None,
# )
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why has this been removed? If I understand correctly from other implementations, this is required, yes?

# compute the previous noisy sample x_t -> x_t-1
latents = self.scheduler.step(noise_pred, t, latents, return_dict=False)[0]

# call the callback, if provided
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were the callback related parts removed here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants